A New Missing Data Imputation Algorithm Applied to Electrical Data Loggers

Nowadays, data collection is a key process in the study of electrical power networks when searching for harmonics and a lack of balance among phases. In this context, the lack of data of any of the main electrical variables (phase-to-neutral voltage, phase-to-phase voltage, and current in each phase and power factor) adversely affects any time series study performed. When this occurs, a data imputation process must be accomplished in order to substitute the data that is missing for estimated values. This paper presents a novel missing data imputation method based on multivariate adaptive regression splines (MARS) and compares it with the well-known technique called multivariate imputation by chained equations (MICE). The results obtained demonstrate how the proposed method outperforms the MICE algorithm.Ministerio de Economía y Competitividad; AYA2014-57648-PAsturias (Comunidad Autónoma). Consejería de Economía y Empleo; FC-15-GRUPIN14-01

A Hybrid Algorithm for Missing Data Imputation and Its Application to Electrical Data Loggers

The storage of data is a key process in the study of electrical power networks related to the search for harmonics and the finding of a lack of balance among phases. The presence of missing data of any of the main electrical variables (phase-to-neutral voltage, phase-to-phase voltage, current in each phase and power factor) affects any time series study in a negative way that has to be addressed. When this occurs, missing data imputation algorithms are required. These algorithms are able to substitute the data that are missing for estimated values. This research presents a new algorithm for the missing data imputation method based on Self-Organized Maps Neural Networks and Mahalanobis distances and compares it not only with a well-known technique called Multivariate Imputation by Chained Equations (MICE) but also with an algorithm previously proposed by the authors called Adaptive Assignation Algorithm (AAA). The results obtained demonstrate how the proposed method outperforms both algorithms.Ministerio de Economía y Competitividad, AYA2014-57648-PAsturias (Comunidad Autónoma). Consejería de Economía y Empleo, FC-15-GRUPIN14-01

Missing data imputation of solar radiation data under different atmospheric conditions

[Abstract] Global solar broadband irradiance on a planar surface is measured at weather stations by pyranometers. In the case of the present research, solar radiation values from nine meteorological stations of the MeteoGalicia real-time observational network, captured and stored every ten minutes, are considered. In this kind of record, the lack of data and/or the presence of wrong values adversely affects any time series study. Consequently, when this occurs, a data imputation process must be performed in order to replace missing data with estimated values. This paper aims to evaluate the multivariate imputation of ten-minute scale data by means of the chained equations method (MICE). This method allows the network itself to impute the missing or wrong data of a solar radiation sensor, by using either all or just a group of the measurements of the remaining sensors. Very good results have been obtained with the MICE method in comparison with other methods employed in this field such as Inverse Distance Weighting (IDW) and Multiple Linear Regression (MLR). The average RMSE value of the predictions for the MICE algorithm was 13.37% while that for the MLR it was 28.19%, and 31.68% for the IDW.Ministerio de Economía y Competitividad; AYA2010-1851

Comparative study of imputation algorithms applied to the prediction of student performance

[Abstract]: Student performance and its evaluation remain a serious challenge for education systems. Frequently, the recording and processing of students’ scores in a specific curriculum have several f laws for various reasons. In this context, the absence of data from some of the student scores undermines the efficiency of any future analysis carried out in order to reach conclusions. When this is the case, missing data imputation algorithms are needed. These algorithms are capable of substituting, with a high level of accuracy, the missing data for predicted values. This research presents the hybridization of an algorithm previously proposed by the authors called adaptive assignation algorithm (AAA), with a well-known technique called multivariate imputation by chained equations (MICE). The results show how the suggested methodology outperforms both algorithms.Ministerio de Economía y Competitividad ; AYA2014-57648-PAsturias. Consejería de Economía y Empleo ; FC-15-GRUPIN14-01

Imputación de datos faltantes en redes de distribución de baja tensión: Aplicación a edificios de pública concurrencia

Tesis doctoral por el sistema de compendio de publicacionesActualmente, la toma de datos es un proceso clave en el estudio de las instalaciones de energía eléctrica; no solo desde un punto de vista de calidad de la distribución, sino también desde un punto de vista económico, así pues, se hace necesario conocer cómo y dónde se consume energía en una instalación. En este contexto, la falta de datos de cualquiera de las principales variables eléctricas objeto de medida (tensión fase-neutro, tensión fase-fase, corriente en cada fase y factor de potencia) en cualquier momento, afecta negativamente al estudio realizado. Cuando esto ocurre, debe realizarse un proceso de imputación de datos para sustituir los datos que faltan por valores estimados. Para el desarrollo de la presente tesis doctoral, se inicia el trabajo demostrando la viabilidad de las técnicas de imputación como herramientas adecuadas para la gestión, monitorización y control de instalaciones de energía eléctrica. Para llevar a cabo dicha demostración se emplean los datos recogidos en una instalación eléctrica tipo (Radiación solar recibida en varias plantas fotovoltaicas de Galicia bajo distintas condiciones atmosféricas) y una serie de algoritmos de imputación de alta eficiencia ya consolidados dentro de otros sectores de actividad como son: Inverse Distance Weighting (IDW), Multiple Linear Regressions Models (MLR Models) y Multiple Imputation by Chained Equations (MICE). En esta primera fase, no solo se demuestra la viabilidad del uso de las citadas técnicas como herramientas capaces de mejorar los sistemas de gestión y monitorización de una instalación, sino que también se evalúa la eficiencia de los citados algoritmos y se selecciona el más eficiente de ellos, MICE, para su uso como elemento de comparación con los futuros algoritmos a desarrollar en la presente tesis doctoral. Posteriormente se presentan los resultados obtenidos al aplicar un nuevo método de imputación en la instalación objeto de estudio (Edificio Severo Ochoa de la Universidad de Oviedo). Este nuevo método, desarrollado, y al que se ha denominado AAA (Adaptive Assignation Algorithm), está basado en el algoritmo inteligente conocido como Multivariate Adaptive Regression Splines (MARS). Tal como ya se adelantó los resultados obtenidos con el nuevo método son comparados con los obtenidos tras la aplicación del algoritmo de referencia MICE demostrándose las ventajas de AAA en términos de precisión y fiabilidad en la tarea propuesta. El estudio pormenorizado del rendimiento de la nueva metodología desarrollada (AAA) demostró que la fiabilidad del nuevo algoritmo decrecía sustancialmente cuando el número de variables faltantes en un mismo registro temporal era muy alto, lo que da pie a buscar nuevas metodologías de imputación, así como la hibridación de las mismas en la fase final del trabajo con objeto de superar la citada limitación. Finalmente, con objeto de obtener imputaciones fiables en situaciones en las que el número de faltantes es muy elevado se desarrolló una nueva metodología basada en redes neuronales auto organizadas SOM (Self-Organized Maps) que se demostró más eficiente en ese tipo de situaciones muy adversas siendo, sin embargo, su rendimiento peor cuando el número de faltantes no es tan elevado. Como la cantidad de faltantes por registro depende en gran medida de problemas en la red y en los equipos de medida y que estos no son a priori predecibles, es posible encontrarse en un corto espacio de tiempo con situaciones donde el número de datos faltantes varía considerablemente de un registro a otro. Por ello, la propuesta final de esta tesis se basa en la hibridación inteligente (o uso selectivo) de los distintos algoritmos desarrollados. Dicha hibridación se ha evaluado haciendo un uso combinado tanto de AAA con MICE como de AAA con SOM, habiéndose mostrado esta combinación como más adecuada independientemente del número de faltantes acontecido. Nowadays, data measurement and collection are key processes in the study of the electric power system. Not only for power quality purposes, but also from an economic point of view, it is necessary to know how and where energy is consumed in a facility. In this context, missing data of any of the main electrical variables under measurement (phase-to-neutral voltage, phase-to-phase voltage, phase current or power factor) may negatively affect the underway study. When this takes place, a data imputation process must be conducted in order to replace the missing data with estimated values. In the development of the present doctoral thesis, the effort is initially directed to prove the feasibility of imputation techniques as appropriate tools for the management, monitoring and control of electric power systems. To carry out this research, a data logger has been used in an electrical installation (global solar radiation received in many photovoltaic plants in Galicia under different atmospheric conditions) and a series of high efficiency imputation algorithms already consolidated within other activity sectors. Among this methods, it is important to highlight the Inverse Distance Weighting (IDW), Multiple Linear Regressions Models (MLR Models) and Multiple Imputation by Chained Equations (MICE). In this first stage of the thesis, the viability of using the aforementioned techniques as tools capable of enhancing the management and monitoring systems of an electrical installation has been demonstrated. Moreover, the efficiency of the said algorithms is evaluated and the one providing the best performance, the MICE, is used as a benchmark for comparison with the new algorithms developed in the present doctoral thesis. Subsequently, the results of applying a new imputation method to the installation under study, the Severo Ochoa Building at the University of Oviedo, are presented. This new method, developed in the present work, and which has been called AAA (Adaptive Assignation Algorithm), is based on an intelligent algorithm known as Multivariate Adaptive Regression Splines (MARS). As already mentioned, the results obtained with the new method are compared with those reached by applying the MICE benchmark algorithm. The advantages of AAA in terms of accuracy and reliability in the proposed task are clearly demonstrated. A detailed study of the performance of the new methodology (AAA) showed that the reliability of the new algorithm decreased substantially when the number of missing variables in the same time register is very high. This fact led to the development of new imputation technologies and their hybridization in the final stage of the work, in order to overcome the aforementioned limitation. The new methodology was developed based on self-organized neural networks SOM (Self-Organized Maps). It proved to be more efficient in very adverse situations being, however, its performance worse when the number of missing variables is not so high. Given that the number of missing data per register depends largely on problems in the network and measurement equipment which are non-predictable or controllable, it is possible to find a short term with situations where the number of missing data changes considerably from a register to another. Therefore, the final proposal of this thesis is based on intelligent hybridization (or selective use) of different algorithms. This hybridization has been evaluated by making a combined use of both AAA with MICE and AAA with SOM. The latter combination results to be the most appropriate regardless of the number of missing events

Student performance prediction applying missing data imputation in electrical engineering studies degree

International Conference on Hybrid Artificial Intelligent Systems, HAIS 2016 (11th. 2016. Sevilla